Skip to content

feat: architecture refactor (microservices), add global configuration, and add experiment system#3

Merged
jvdtoorn merged 11 commits intomasterfrom
refactor-services
Nov 25, 2025
Merged

feat: architecture refactor (microservices), add global configuration, and add experiment system#3
jvdtoorn merged 11 commits intomasterfrom
refactor-services

Conversation

@jvdtoorn
Copy link
Collaborator

What changed

This PR refactors OpenDT from a bit of a monolithic app into a microservices setup. The codebase is now organized into four separate Docker services (dashboard, dc-mock, sim-worker, kafka-init) plus a shared library for common code. Each service has its own Dockerfile and runs independently. All services have documentation with READMEs explaining how they work. The shared library (libs/common/) contains all the Pydantic models, Kafka utilities, and config management code that multiple services need.

I also added a centralized config system using YAML files instead of using in-code constants at several places. There's a main config/default.yaml for global settings and separate configs for different experiments in config/experiments/. This makes it easier to manage different simulation scenarios.

A new feature I added is the experiment management system. You can now run experiments with a simple make experiment name=experiment_1 command, and it automatically handles output directories, result tracking in Parquet files, and generation of plots comparing actual vs simulated power consumption. There's also a debug mode that saves all the OpenDC input/output files if you need to troubleshoot.

The simulation engine got some nice upgrades. It now caches results to avoid re-running identical simulations and uses heartbeat messages for reliable window processing.

Finally, I added a Makefile with commands for everything - make up to start services, make logs-sim-worker to view logs, make experiment to run experiments, etc. Services support hot reload during development, so you don't need to rebuild containers every time you change code.

@jvdtoorn jvdtoorn merged commit 534b42e into master Nov 25, 2025
2 checks passed
@jvdtoorn jvdtoorn deleted the refactor-services branch November 25, 2025 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant